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ABSTRACT 


Conventionally programmed digital computers can process numbers with 
great speed and precision, but do not easily recognize patterns or imprecise 
or contradictory data. Instead of being programmed in the conventional 
sense, artificial neural networks (ANNs) are capable of self-learning through 
exposure to repeated examples. However, the training of an ANN can be a 
time consuming and unpredictable process. 

A general method is being developed by the author to mate the adapt- 
ability of the ANN with the speed and precision of the digital computer. 
This method has been successful in building feedforward networks that can 
approximate functions and their partial derivatives from examples in a single 
iteration. The general method also allows the formation of feedforward net- 
works that can approximate the solution to nonlinear ordinary and partial 
differential equations to desired accuracy without the need of examples. It is 
believed that continued research will produce artificial neural networks that 
can be used with confidence in practical scientific computing and engineering 
applications. 
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INTRODUCTION 


Artificial neural networks (ANNs) have proven to be versatile tools for ac- 
complishing what could be termed higher order tasks such as pattern recog- 
nition, classification, and visual processing. However, conventional wisdom 
has held that networks are unsuited for use in more purely computational 
tasks, such as mathematical modelling and physical analysis of engineering 
systems. Certainly the biological underpinnings of the neural network con- 
cept suggest that networks would perform best at tasks at which biological 
systems excel, and worse or not at all at other tasks. 

Contrary to opinion the author believes that continued research into 
the approximation capabilities of networks will enable the neural network 
paradigm, with all of its advantages in behavior and adaptability, to be 
mated to the more purely computational paradigms of mathematically ori- 
ented scientific programming and analysis. Additionally, it is felt that the 
thorough investigation of network approximation capabilities will benefit the 
network field and connectionism in general. 

In a field as conceptually difficult as the study of artificial neural networks, 
it is best to start investigation with supervised learning, test the established 
premises, and alter them to circumvent pitfalls in implimentation. 

FUNCTION APPROXIMATION 
Learning as Function Approximation 

Central to the author’s research approach is the view that supervised learning 
in artificial neural networks is equivalent to the problem of approximating 
a multivariate function and that learning should be able to be explained 
by approximation theory. Approximation theory deals with the problem of 
approximating or interpolating a multivariate function. This approach has 
been considered by other researchers in the field of ANNs [l]-[4]. However, 
the author extends this assumption of function approximation by assuming 
that ANNs can model discontinuous multivariate functions and should be at 
least as accurate and numerically efficient as existing computational tech- 
niques used in science and engineering. Also, ANN behavior and adaptation 
difficulties, from supervised learning to machine vision, should be amenable 
to the standard error analysis techniques used in numerical analysis. 

In a general sense experiments, analytical methods, and computational 
methods can be considered to be forms of function approximation. The 
governing equations derived from analytical methods are a compact repre- 
sentation of the functions that model some particular phenomena observed 
in experiments. Computational techniques are used to approximate the func- 
tion or functions that satisfy the governing equations. The graphs and tables 
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made from experiments are representations of the functions that underlie ob- 
served physical phenomena. 

PROGRAMMABLE ARTIFICIAL NEURAL NETWORKS 

If we are to assume that ANNs are as valid as established computational tech- 
niques, then ANNs should be evaluated in the same manner as are computa- 
tional techniques. In evaluating the capabilities of a new numerical method, 
it is prudent to first apply it to the solution of algebraic and ordinary and 
partial differential equations of known behavior. This same approach can be 
used for ANNs since the solution of algebraic and differential equations can 
be viewed as the approximation of a function that must satisfy the equation 
in question subjected to boundary and/or initial conditions. 

Applying an ANN to the solution of an algebraic or differential equation 
effectively uncouples the influences of the quality of data samples, network 
architecture, and transfer functions from the network approximation perfor- 
mance. The solution of equations also allows us to study the influence of 
constraining the connection weights. The most immediate benefit in this 
approach would be the construction of networks that can approximate the 
solution to desired equations without the need for examples. This would be 
of value in engineering applications since considerable effort may be saved 
if the equations governing a physical process can be directly incorporated 
into the neural network architecture without the need of examples, thereby 
shortening or even eliminating the learning phase. 

The author has previously reported the use of the piecewise linear hard 
limit transfer function [5] in solving model algebraic and linear ordinary 
differential equations by the feedforward architecture [7]. In this report the 
author will use the recurrent artificial neural network (RANN) architecture to 
model a chaotic system by solving a nonlinear ordinary differential equation 
known as Duffing’s equation. 


APPROACH 

A simple RANN consists of two layers of processing elements as illustrated 
in Fig. 1 for two coupled time-dependent variables u and v, where 7 denotes 
the input layer, and 77 denotes the output layer. Generally speaking, the 
inputs u and u, fed into layer I at time t n , are operated on by the processing 
elements (PEs) and then multiplied by the constant connection weights a iq . 
The coefficient a xq is the connection weight between the ith neuron of layer 7 
and the <?th neuron of layer II. As a result, the outputs of layer 7, multiplied 
by the connection weights, are input to layer 77 PEs. The output of layer 77 
neurons are the values of the dependent variables at time t n+1 . These values 
are in turn fed back into the input layer 7 for the next iteration. 
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The dynamic behavior of such a RANN is governed by the value of the 
connection weights, the type of processing elements in the RANN, and the 
transfer functions used. In this study we restrict ourselves to using the 
piecewise linear transfer function in all layers of the RANN, since it is one of 
the simplest functions to implement on both digital and analog forms [8]. 

The hard limit function T^(u) (Fig. 2) operating on the dependent vari- 
able u for the ^th neuron of the Kth. layer can be modelled by the following 
equations: 

f -1.0 for £ q < -1.0 
T*(u)=< for -1-0 < < 1.0 

+ 1.0 for £ q > 1.0 

where 

^ a iq u t for additive neurons 
» 

= JJ diqUi for multiplicative neurons 

i 

By correctly assigning different values to the RANN connection weights, 
one is able to integrate a system of coupled recurrent relations into the archi- 
tecture without the need of training. We will now show how to obtain such 
recurrent relations from a system of ordinary differential equations. 


Duffing’s Equation 

The inhomogeneous Duffing’s equation describes the forced motion of a par- 
ticle between two equilibrium states and can be written as the following 
nondimensional second-order ordinary differential equation 

d?x dx 1 / 

W + 2fi dt~2 ( x-x ) =Fo cos ^ (!) 

where t , x, /i. F 0 . and u) represent time, displacement, damping coefficient, 
force amplitude, and frequency of excitation respectively. The importance of 
this equation is that its chaotic and nonchaotic behavior has been extensively 
examined by theoretical [9], experimental [10], and numerical methods [6], 

[11] -[13]- ‘ 

In order that Duffing’s equation can be more easily approached by the 
RANN programming method and so that we may obtain particle displace- 
ment, velocity, and acceleration, Eq.(l) is reduced to a system of two first- 
order equations by the following change of variables 



1 dx 1 . 1 cPx k 2 dy 

K 2 dt K 2 /C3 dt 2 K3 dt 
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where K\, k 2 , and /c 3 are constants used to scale the values of the new vari- 
ables. Therefore, Eq.(l) becomes 


ds 

dt 

dy_ 

dt 



( 2 ) 

( 3 ) 


where /c 4 = — . 

It will now be demonstrated how the RANN architecture can be used to 
approximate the solution to Eq.(2) and Eq.(3) for arbitrary coefficients. 


Integration of Buffing’s Equation 

For this problem the initial value problem of Eqs.(2) and (3) will be inte- 
grated by the explicit MacCormack method [14] which is a finite-difference, 
predictor-corrector scheme commonly used in the solution of time-dependent 
fluid dynamics equations. The application of the MacCormack technique to 
a general first order ordinary differential equation 

du 

— - /(«,<) = 0 , 

where / is some arbitrary function of the dependent variable u and indepen- 
dent variable t, results in 

Predictor: u* = u n + A tf(u n ,t n ) 


Corrector: u" +1 = 1 (u n + u* + A tf(u*,t*)) = u n + — (f(u n ,t n ) + 

where the superscripts denote the time level and t* = t n + A t. 

Application of the MacCormack method to Eqs.(2) and (3) results in 

s* = 8 n + Atf 1 {s n ,y n ,t n ) 


„n+l 


= s" + y(/i(3 n ,y n d n ) + /i(/,!/*,<*)) 
y* = y n + Atf 2 (s n ,y n ,t n ) 

y n+1 = y n + Y (/2(sn ’ s ' n,r) + /2(5%2/ * ,r)) 


where 


1 F 

Ms,y,t ) = K A y and / 2 (s, y, t) = -2fiy + — (s - ki 2 s 3 ) + — cos (ut) 

v ' K 2 
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Substitution of the expressions for /*, / 2 , s*, and y* into the equations for 


s n+1 and y n+l 

p 

results in the following algebraic system, 

= (s n ) 3 

(4) 

r n 

= (s *) 3 = (s n + K A At y n f 

(5) 

s n + l 

= His n + f/ 2 t/ ri + # 3 p n + #4 cos (ruuAt) 

( 6 ) 

y n+ 1 

= H 5 s n + Hey n + #Vp n + H & r n + #9 cos (nu/At) + 
#io cos ((n + l)aAf) 

(7) 


The coefficients #1 through H\q are 

At 2 


#1 = 1 + — — , #2 = K 4 At (1 — flAt ) , #3 = 


K 2 At 2 


H 4 = 


F 0 At 2 


A f A / 2 

* , #5 = r— (i - , #6 = 1 - 2pAt + 2p 2 At 2 + — 

2 /c i 2 k ,4 4 


«i 3 A/ v /Cj 3 F 0 zXi 

#7 = — (1 — 2/xAt) , #8 = — , #9 = r-(l - 2/j.At ) 

k 2 4 /c 2 4 k 2 2 

Hl0 = ^Y ’ (8) 

/v2 i 


If At is required to be constant, then the time-dependent coefficients of Eq.(8) 
become constants for specific values of //, F 0 , «i, /c 2 , and /c 3 . Equations (4) 
through (7) are the linear and nonlinear algebraic equations that approximate 
Duffing’s equation and must be modelled by the RANN. 


Network Approximation of Duffing’s Equation 

To compare against previous numerical studies of Duffing’s equation, we 
require only that the RANN generate values for particle displacement and 
velocity. Figures 3 - 5 show the connections required in a constrained RANN 
modelling Eqs.(4) through (7) respectively. It is understood that the network 
output is fed back to the input layer though the connections are not shown 
in the figures. An initial time f° = 0 is assumed. 

Note that the magnitude of all inputs and outputs are scaled to be less 
than value 1 so that hard limits may be used in the processing elements. 
This constraint on the magnitude of the dependent variables s n , y n , p n , and 
r n requires that 

/Ci |x| TOOX + ^kt 

Equations (4) and (5) possess cubic time-dependent unknowns that are 
approximated by the RANN using the multiplicative neuron model also 
known as the Pi neuron (Fig. 3) [5]. In traditional ANNs, the Pi neu- 
rons are used to achieve greater processing power by using more complex 
connections. 


dx 

dt 


k 2 > 


dx 

dt 


«3 > 


cPx 


dt 2 


(9) 
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The fully assembled RANN is shown in Fig. 6. Updated values of the 
cosine functions ( cos (nuAt) and cos ((n + l)u;At) ) must be input at each 
iteration in the RANN. 


RESULTS 

The RANN of Fig. 6 was constructed and run with a fixed damping value 
of n — 0.084 and fixed forcing amplitude of F 0 = 0.178. Previous results 
of the two-well potential system [6] indicate displacement ranging between 
± 2.0 and velocity ranging between ± 1.0. Assuming that At < \x\ max Eq.(9) 
allows the use of k i = 4.0, k 2 = 2.0, and k 3 = 1.0. Initial conditions were 
set for x(0) = 1.0 and x(0) = 0.0, except for Fig. 8 where x(0) = 0 and 
x(0) = 1.0 for the coexisting global attractor. To ensure that steady-state 
solutions were displayed, approximations were run for 900 periods, T, based 
on the forcing frequency u (T = 27 rju). All computations were done in 
double precision and all test cases were run in less than 40 real time seconds 
on a Sparc 10 SX Model 512 workstation. 

RANN Simulation Results 

Figures 7-9 compare the constructed RANN output (x = k^s and x = /c 2 y) 
with the numerical results of Masoud and Asfar [6], for the same parame- 
ter values and initial conditions. Here, as with reference [6] the damping 
and excitation amplitude are based on the work presented by Pezeshki and 
Dowell [13]. Throughout the simulation the time step used in the RANN 
was identical to that used in the Masoud and Asfar study, At = T/100. For 
clarity, plots of time histories have been excluded. Phase plane trajectories, 
as well as Poincare maps are shown and compared in cases where they are 
shown by reference [6]. 

Figure 7 starts with u = 1.020 at which the period of the oscillator is 
equal to 1 (period 1 oscillation) and doubles as u decreases. Thus we ob- 
serve period 2, period 4, and period 8 oscillations, until the system evolves 
into a coexisting global period 5 motion (Fig. 8) at which the system oscil- 
lates between the two equilibrium states (x, x) respectively (0,-1) and (0, 1). 
Figure 9 describes the first region of chaotic oscillations starting at u — 0.865 
and ending at ui = 0.780. 


CONCLUSIONS 

Using the piecewise linear transfer function and additive and multiplicative 
neurons the author was able to construct a recurrent artificial neural network 
that was capable of accurately solving a complex nonlinear problem. The 
solution of the inhomogenous Dufling’s equation was shown as a numerical 
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example and compared well with the computational results of Masoud and 
Asfar. This was done without the need for data or conventional training. By 
establishing a direct link between a well known numerical method and the 
operation of a RANN we believe it may be possible to link common numerical 
methods from computational mechanics to the operation of the more popular 
ANN paradigms. 
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Figure 1: Basic RANN architecture. 



Figure 2: Hard limit transfer function. 
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Figure 3: RANN representation of the 
nonlinear equations for p n and r n . 
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Figure 4: RANN representation of the 
recurrent equation for s n+1 . 




Figure 5: RANN representation of the 
recurrent equation for y n+1 . 
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Figure 6: Full RANN assembly for the 
MacCormack scheme applied to Duff- 
ing’s equation. 




Figure 8: Phase space trajectories from 
Masoud and Asfar (top set) and the 
RANN (bottom set). From left to right: 
coexisting (a) u = 0.957 period 8, (b) 
u> = 0.957 global period 5. 


Figure 7: Comparison of phase space 
trajectories from Masoud and Asfar 
(top set) and the RANN (bottom set). 
From left to right: (a) u = 1.020 period 
1, (b) to = 1.000 period 2, (c) u> = 0.963 
period 4. 
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VELOCITY VELOCITY 



Figure 9: Comparison of phase space 
trajectories and Poincare maps in the 
first chaotic region from Masoud and 
Asfar (top set) and the RANN (bot- 
tom set). From left to right: (a) u> = 
0.865 chaotic higher frequency bound- 
ary, (b) ij = 0.780 chaotic lower fre- 
quency boundary. 
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